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ABSTRACT 

During  constructive  and  virtual  training  and  experimentation,  circumstances  exist  when  human  intervention  is  required  to 
override  the  outcomes  of  the  computer  simulation  or  to  change  the  state  of  the  simulation.  Some  examples  of  human 
intervention  include  adjudication  of  engagements  having  unreasonable  outcomes,  changing  supply  levels,  and  moving  units 
or  individual  entities  within  the  battlespace.  Overriding  the  simulation  results  during  event  execution  should  not  be  a 
spontaneous  decision.  If  done  randomly  or  without  established,  well  briefed  procedures,  the  training  audience  or  experiment 
participants  will  lose  confidence  in  the  validity  of  the  computer  model.  During  event  execution,  the  results  of  human 
intervention  can  cause  confusion  on  the  part  of  participants,  give  unfair  advantages  (perceived  or  actual)  to  one  side  or 
another  and  corrupt  data  collection.  Additionally,  these  interventions  must  be  performed  in  a  timely  manner  to  have  their 
desired  effect.  For  these  reasons,  management  must  implement  well  reasoned  adjudication  procedures.  For  the  2011  Joint 
Forcible  Entry  Warfighter  Experiment  (which  included  US  Army,  Air  Force,  Navy  and  Marine  forces  operating  in  a 
distributed,  constructive  simulation  over  two  weeks),  the  Experimentation  Environments  Branch  of  the  Army’s  Maneuver 
Battle  Lab  developed  a  set  of  procedures  to  provide  guidance  when  simulation  outcomes  will  be  changed  or  when  human 
intervention  into  the  simulation  is  appropriate.  This  paper  provides  analysis  from  the  execution  of  the  experiment  of  how 
these  procedures  impacted  experiment  objectives  and  post-experiment  data  analysis.  Discussion  includes  how  data  collected 
from  the  High  Level  Architecture  (HLA)  logger  and  Damage  Effects  Server  was  used  to  make  intervention  decisions  and 
reasons  for  human  override  of  the  simulations.  The  paper  closes  with  lessons  learned  during  this  event  that  can  be  applied  by 
others  in  the  training  and  experimentation  community  of  practice. 
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INTRODUCTION 

The  purpose  of  this  paper  is  to  present  a  methodology 
for  determining  when,  and  under  what  conditions, 
humans  should  reverse  the  outcomes  of  computer 
simulations  through  an  adjudication  process.  From  the 
outset,  it  is  important  to  note  that  adjudications  or 
changes  made  to  the  outcomes  of  computer  simulations 
can  never  be  done  to  influence  results  or  change 
conditions  in  order  to  influence  analytical  outcomes. 
Readers  are  encouraged  to  adapt  the  procedures 
outlined  in  this  paper  for  their  own  experimentation 
and  training  events.  It  must  be  remembered  that  key 
components  of  the  process  are  the  determination  of 
procedures  prior  to  event  execution  and  the  full 
accounting  of  adjudication  is  recorded  in  an  open 
process. 

It  is  said  that  all  models  are  imperfect.  When  using 
models  and  simulations  for  training  and 
experimentation,  leaders  must  develop  procedures  to 
account  for  shortcomings  in  the  models.  These 
procedures  cannot  be  arbitrary;  they  must  be 
predictable  and  understood  by  all  participants.  In  most 
training  and  experimentation  events,  leaders  simply 
decide  how  to  account  for  model  shortcomings  as  they 
arise  in  execution.  Although  this  technique  is  perceived 
to  be  the  most  expedient  manner  to  resolve  problems, 
in  practice  there  is  normally  time  to  list  the  decision 
making  process,  which  may  affect  experiment  or 
training  outcomes.  Additionally,  in  most  cases  the 
procedures  are  inconsistent,  undocumented,  and  result 
in  reduced  confidence  on  the  part  of  the  participants  in 
the  simulation  or  model.  This  paper  has  four  purposes. 
First,  to  substantiate  the  requirement  for  adjudication 
procedures  for  all  training  and  experimentation  events. 
Second,  to  present  a  framework  for  establishing 
adjudication  procedures.  Third,  to  identify  the 
procedures  for  executing  the  procedures  and  who 
should  execute  them.  Fourth,  to  present  lessons  learned 
from  using  the  procedures  dining  Joint  Forcible  Entry 
Warfighter  Experiment  (JFEWE). 


In  order  to  illustrate  the  concepts  that  I  will  outline  in 
this  paper,  I  will  use  the  March  2011  JFEWE  as  an 
example  of  how  these  procedures  were  used,  and  will 
illustrate  both  good  and  bad  examples  of  the  use  of  the 
procedures  during  that  experiment  and  the  lessons 
currently  being  learned  from  their  use. 

JFEWE  Construct 

As  summarized  in  the  JFEWE  Final  Report,  JFEWE 
was  an  experiment  held  in  March,  2011.  It  involved 
496  participants  directly  associated  with 
experimentation  execution.  It  was  an  Army  Concepts 
and  Integration  Directorate  (ARCIC)-directed, 
integrated  warfighting  experiment  with  the  Maneuver 
Battle  Lab  (MBL)  assigned  as  the  executive  agency  for 
planning  and  execution.  The  JFEWE  was  a  concept 
development  constructive  and  virtual  simulation 
experiment  focused  on  Forcible  Entry  (FE)  doctrine 
and  concepts.  Its  goal  was  to  examine  Joint  and  Army 
capability  to  conduct  FE  operations  by  amphibious 
assault,  airborne  assault,  and  air  assault  into  austere 
locations  to  overcome  or  avoid  enemy  anti-access  and 
area  denial  efforts,  and  to  present  the  enemy  with 
multiple  threats  from  unexpected  locations.  JFEWE 
provided  insights  to  the  FY  11/12  Army  Warfighting 
Challenges  (WfC)  and  to  the  development  of  the  Joint 
Operational  Access  Concept  (JOAC)  and  Army 
Operational  Concepts  (AOC).  This  effort  provided 
concept  writers  and  doctrine  developers  with  feasibility 
feedback  for  the  refinement  of  their  Functional 
Concepts  as  well  as  identifying  required  capabilities  in 
the  2018  time  frame,  which  informed  Doctrine, 
Organizational,  Material,  Training,  Personnel  and 
Facilities  (DOTMLPF)  integrated  solutions  across 
warfighting  functions.  The  chart  from  the  Final  Report 
on  page  2  shows  the  overall  JFEWE  Federation  and  the 
functions  of  each  location. 
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JFEWE  Experiment  Objectives 

The  Maneuver  Battle  Lab,  in  conjunction  with  the 

community  of  practice,  identified  the  following 

ARCIC-approved  Experiment  Objectives: 

1 .  Focus  on  the  operational  problem  of  Anti-Access 
with  a  significant  tactical  force  (Multiple 
BDEs/DIV  force). 

2.  Critically  explore  and  assess  concept  supporting 
ideas  and  required  capabilities  in  challenging 
operational  environments  to  revise  Army 
concepts  to  drive  capability  development  in  the 
2018-2026  timeframe. 

3.  Economize  the  Army  Experimentation  Plan  by 
integrating  events  using  a  consistent  campaign 
scenario  and  consistent  Models  and  Simulations 
Federation. 


4.  Inform  US  Army  Forces  Command  contingency 
force  operations  requirements  and  capabilities. 

5.  Incorporate  Division  Command  Posts  play  into 
the  experiment. 

6.  Conduct  the  experiment  within  a  joint 
context/problem  to  inform  and  exercise  Joint 
Forces  Command’s  (JFCOM)  draft  JOA  Concept. 

7.  As  part  of  the  same  complex  environment  utilize 
the  same  or  similar  scenarios  in  the  various 
venues. 

8.  Develop  a  campaign  plan  context  for  forcible 
entry  employing  tactical  forces  (Multiple 
BDEs/DIV)  with  Division  and  Corps  Command 
and  Control  (C2)  requirements. 


Community  of  Practice  (CoP) 


Mission  Command  CoE,  Ft  Leavenworth,  KS 
JTF,  JFLCC,  JFMCC,  JFACC  &  DIVHQs Staff  -  (107) 


SCoE,  Ft  Lee,  VA 


TRADOC  G-2,TRISA, 
Ft  Leavenworth,  KS 


Space  and  Missile  Defense  Center, | 
Colorado  Springs,  CO  & 

Redstone  Arsenal,  AL 
H  ICON  (6) 

JEMS&  EADSIM 

Intel  CoE,  Ft  Huachuca,  AZ 


☆ 


Legend 

Support  Site 


Distributed  Site 


FCoE,  Ft  Sill,  OK 
FIRES  BDE  (56) 

FIRESIM,  EADSIM  &  RTOS 


AMEDD,  Ft  Sam  Houston,  TX 


ARCIC,  Ft  Monroe,  VA 


USAF,  505th  Cmd  and  Control 
WingHurlburt  AFB,  FL 
Various  USAF  Squadrons  (28) 

AWSIM  Aviation  CoE,  Ft  Rucker,  AL 

AVNBDE  (39) 

A  TCOM  &  A  CRT- A  V 


USN  Warfare  Development 
Command,  Newport,  Rl 

USMC,  Warfighting  Laboratory, 
Quantico,  VA 


JOINTJ9,  Suffolk,  VA 

USAF,  Air  Component 
k  Command,  Langley  AFB  VA 
USAF  Experimentation 
Office,  Hampton,  VA 


ARSOCIC,  FT  Bragg,  NC 
JSOTF  and  IBCT  Bde  Staff-  (10) 

XVIII  ABC,  Ft  Bragg,  NC 


Signal  CoE,  FT  Gordon,  GA 
S6  at  each  maneuver  unit  -  (20) 

MCOE,  Ft  Benning,  GA 

IBCT,  R&S  Bde,  A/CO  SBCT&  USMC  MEB  -  (215) 
OneSAF&ACRT-GV 


Total  of  496  personnel  participated  in  JFEWE  SIMEX 


Figure  1.  Community  of  Practice  Graphic 


Due  to  the  size  and  complexity  of  the  experiment, 
planners  knew  early  on  in  the  experiment  design 
process  that  participants  would  require  a  process  to 
follow  in  the  event  of  simulation  or  human  error. 


Importance  of  Establishing  Adjudication 
Procedures 

Many  factors  can  lead  to  the  need  for  human 
intervention  into  the  outcomes  of  the  models.  The  most 
common  examples  are  unreasonable  effects  (a  bullet 
fired  from  a  pistol  destroys  a  tank),  unrealistic 
representation  (an  entity  represented  as  a  tank  actually 
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models  a  pickup  truck),  terrain  not  modeled  correctly 
(a  bridge  which  should  allow  traffic  to  pass  at  normal 
speed  across  a  river  stops  traffic),  and  when  White  Cell 
[the  Experiment  or  Exercise  Control  Cell]  intervention 
is  required  (an  enemy  unit  requires  movement  to  a 
specific  point  faster  than  it  would  normally  move  to 
ensure  a  scripted  event  occurs).  Additionally,  issues 
involving  the  inability  of  simulations  to  properly 
integrate  with  one  another  may  require  human 
adjudication.  For  example,  an  aircraft  in  one 
simulation  can  sense  ground  forces  in  a  second 
simulation  as  expected,  but  the  ground  forces  cannot 
sense  the  aircraft  even  though  they  should  have  this 
ability.  Given  available  time,  integration  issues  should 
be  resolved  prior  to  event  execution,  but  this  is  not 
always  possible.  Reasons  for  delay  in  integration  may 
include  limited  time,  technical  constraints,  or  budget 
limitation.  If  the  integration  shortcomings  are  minor  or 
will  not  affect  experiment  objectives,  it  may  be 
possible  to  work  around  the  problems  using 
adjudication  procedures.  Identification  of  the  most 
possible  situations  which  will  require  adjudication  is 
the  first  step  in  the  establishment  of  adjudication 
procedures. 

Framework  for  the  Establishment  of  Adjudication 
Procedures 


The  following  are  the  six  steps  to  develop  adjudication 
procedures: 


Identify  known  conditions  which  trigger 
adjudication  procedures 

Identify  members  of  the  Ad  judication  Team 

Develop  stepsto  correct  the  outcome 
in  the  simulation 

B 

•w 

Develop  tracking  procedures  for  each 
adjudication  request 

Implement  the  ad  judication  procedures 

▼ 

Conduct  after  action  review  and  update  procedures 

Figure  2.  Steps  of  the  Adjudication  Procedure 


Step  1:  Identify  Known  Conditions  which  Trigger 
Adjudication  Procedures 

There  can  be  numerous  circumstances  that  trigger  the 
use  of  the  adjudication  procedures.  It  is  important  to 
identify  as  many  potential  situations  as  possible  in 
order  to  outline  the  contingencies  and  procedures  for 
each  set  of  circumstances.  Like  all  contingencies,  the 
mere  discussion  of  the  contingencies  ahead  of  time 
facilitate  resolution  when  these  conditions  surface  and 
allow  for  quicker  and  ideally  better  decisions  dining 
execution.  Consider  activities  such  as  fire  evacuation 
plans  and  giving  these  presentations  in  front  of 
audiences.  Regardless  of  how  good  the  plans  or 
presentations  are,  it  takes  practice  for  them  to  be 
effective.  As  previously  discussed,  the  most  common 
situations  involve  modeling  errors.  They  may  also 
involve  secondary  effects  of  those  errors.  One  such 
example  would  be  if  a  rifle  shoots  two  rounds  of 
ammunition  for  each  "trigger  pull”  ammunition  will  be 
expended  twice  as  fast  as  it  should  be.  Procedures  may 
dictate  that  either  resupply  of  one  basic  load  can  be 
done  once  it  has  been  reported  (further  discussion  in 
Step  4)  or  that  the  model  begins  the  fight  with  twice  its 
basic  load  of  ammunition.  Situations  may  exist  where 
weapons  effects  are  not  reasonable,  or  environmental 
effects  are  not  replicated  accurately  (terrain,  weather 
and  light  data,  buildings  and  other  structures  offer  no 
protection  or  too  much  protection,  sensors  perceiving 
inaccurately,  etc.).  The  important  point  is  to  brainstorm 
and  identify  as  many  contingencies  as  possible  in  order 
to  develop  the  rules  for  each  well  ahead  of  time. 

Step  2:  Identify  Members  of  the  Adjudication 
Team 

The  executive  agent  for  the  experiment  or  training 
event  must  determine  who  will  have  the  authority  to 
execute  the  adjudication  procedures.  Consider  this  your 
umpire  team  or  officiating  crew.  They  should  be 
members  of  the  experiment  White  Cell;  that  is  to  say 
they  are  impartial  and  do  not  favor  either  the  friendly 
or  enemy  forces.  That  said,  there  must  be 
representation  from  both  the  friendly  and  enemy  forces 
in  order  to  ascertain  the  ground  truth  of  an  event.  The 
friendly  and  enemy  representatives  on  the  adjudication 
team  should  research  each  request  in  order  to  validate 
the  request  and  ensure  conditions  and  events  occurred 
as  reported.  During  an  experiment  at  the  MBL,  a  blue 
role  player  requests  an  adjudication  to  reconstitute  15 
members  of  his  unit  because  "an  enemy  soldier 
appeared  out  of  nowhere  and  began  shooting."  When 
the  enemy  forces  were  asked  about  the  incident,  they 
explained  that  the  soldier  who  killed  the  blue  forces 
had  infiltrated  the  friendly  position  over  the  course  of 
approximately  four  hours,  undetected  by  blue  forces.  In 
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such  situations,  one  member  of  the  team  must  be 
knowledgeable  on  the  technical  aspects  of  the  models 
and  simulations  being  used  plus  any  issues  with 
integration  of  different  models.  This  team  member  will 
know  if  "unreasonable  effects"  reports  are  attributed  to 
the  model  or  not.  Additionally,  this  person  may  be  able 
to  give  a  technical  answer  to  experiment  participants  as 
to  why  effects  were  not  necessarily  unreasonable.  For 
example,  the  probability  of  hit  or  probability  of  kill 
calculations  may  be  much  different  than  the 
participants  think  that  they  should  be.  The  leader  of  the 
Adjudication  Team  must  be  someone  knowledgeable 
with  the  real  world  system  capabilities  involved  in  the 
experiment,  serve  as  the  ultimate  arbiter  between  the 
friendly  and  enemy  representatives,  and  balance  their 
input  with  the  input  from  the  technical  representative. 
A  member  of  the  Analysis  Team  must  be  on  the 
adjudication  team  to  ensure  that  input  is  received  for 
each  request  so  that  the  impact  on  the  Experiment 
Objectives  is  part  of  the  discussion  of  any  adjudication. 
The  result  in  this  appointment  is  as  questions  arise 
during  post  experiment  processing,  a  member  of  the 
Analysis  Team  can  answer  why  specific  adjudications 
were  approved  or  disapproved.  The  final  member  of 
the  team  is  the  recorders  who  receives  all  requests,  logs 
them,  gives  to  the  team  for  decision  and  records  the 
actions  taken  or  not  taken  regarding  each  request. 

Step  3:  Develop  Steps  to  Correct  the  Outcome  in 
the  Simulation 

For  each  of  the  expected  types  of  adjudications, 
procedures  must  be  identified  that  will  be  taken  every 
time  that  event  occurs.  Consistency  and  validity  of  data 
as  well  as  the  need  for  standardization  require  this.  For 
example,  if  an  aircraft  of  Paratroopers  is  destroyed  en 
route  to  the  drop  zone  due  to  unreasonable  (meaning 
too  effective)  enemy  air  defense  fire,  there  must  be 
procedures  for  how  those  Soldiers  will  be 
reconstituted,  at  what  location  they  will  come  back  to 
life,  and  if  they  should  remain  casualties.  If  four 
vehicles  are  destroyed  due  to  a  convoy  entering  a 
minefield,  and  it  is  determined  that  under  normal 
tactical  situations,  one  or  perhaps  two  vehicles  would 
be  destroyed  and  the  third  and  fourth  vehicles  are 
reconstituted,  it  must  be  identified  whether  the  third  or 
fourth  will  be  reconstituted  "fresh",  with  its  full  load  of 
ammunition  and  fuel  or  whether  that  will  be 
decremented  to  account  for  where  it  was  in  the  fight 
when  they  entered  the  minefield.  The  same  applies  for 
the  crew  of  the  vehicles.  Depending  on  the  fidelity  of 
the  simulation,  the  adjudication  team  may  have  to 
decide  on  the  amount  of  food  and  water  to  reconstitute. 
With  some  simulations,  when  reintroducing  an  entity, 
they  must  come  back  at  100%  strength.  Adjudication 
Team  members  should  develop  standardized 


procedures  for  as  many  situations  as  possible  to 
accelerate  the  process  during  execution  and  ensure 
consistency.  The  following  chart  represents  the  general 
process  used  dining  JFEWE  and  was  used  to  brief 
leaders  at  the  Final  Planning  Conference. 


Un  reasonable  Event  Occurs  or  Simulation  error 

Federate  EXCON  Investigates  and  makes 
recommendation  to  White  Cell  using  Adjudication  Request 

White  Cell  investigates  with  Red  Commander, 
and  all  involved  EXCONs 

White  Cell  adjudicates  event 

White  Cell  Informs  AnalysisCell,  Federate 

EXCONs  involved,  and  Federation  Battlemaster 
o  f  ad  j  ud  i  cation  d  ec  isl  on 

If  approved,  Federation  Battl  emaster  in  forms  DES 
Operatorto  execute  adjudication 

DES  Operator  informs  Federation  Battlemaster 
when  adjudication  is  complete;  Federation  Battlemaster 
informs  EXCON  when  complete 

Figure  3.  Adjudication  Flow 

Step  4:  Develop  Tracking  Procedures  for  Each 
Adjudication  Request 


In  order  to  maintain  the  validity  of  the  experiment  and 
credibility  of  the  data  collection  process,  it  is 
imperative  that  all  requests  and  actions  taken  are 
documented.  Detailed  tracking  procedures  usually 
begin  with  the  simulation  interactor  or  military  role 
player,  either  friendly  or  enemy.  When  they  observe  an 
action  they  feel  warrants  adjudication,  a  process  must 
exist  for  them  to  submit  a  detailed  request  with  all 
pertinent  information  of  the  conditions  surrounding  the 
event.  A  vague  report  such  as  ‘‘our  guys  keep  receiving 
indirect  fire  at  a  volume  far  in  excess  of  enemy 
capabilities”  is  almost  impossible  to  investigate  or 
rationalize.  Likewise  “one  of  our  tanks  just  ran  out  of 
fuel  and  should  have  had  plenty  of  fuel”  does  not  give 
sufficient  detail  to  warrant  adjudication.  As  a 
minimum,  the  Adjudication  Team  will  need  the 
following:  The  exact  time  or  as  close  as  possible, 
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entities  involved,  location  stated  as  accurately  as 
possible,  and  what  exactly  is  to  be  adjudicated.  For 
example,  if  a  personnel  carrier  and  all  personnel  on 
board  were  destroyed,  it  is  possible  that  the  requested 
adjudication  is  merely  regeneration  of  some  of  the 
personnel  on  board,  not  the  entire  number  of  soldiers 
killed.  In  order  to  facilitate  the  process,  the  MBL 
developed  a  tracking  sheet,  the  adjudication  “buck 
slip,”  that  each  simulation  and  military  role  player  fills 
out  to  begin  the  process.  Figure  4  is  an  example  of  the 
buck  slip  used  during  JFEWE  by  the  simulation 
interactors  and  military  role  players. 


ADJUDICATION  REQUEST 

DTG  OF  SIM  EVENT: 

REQUESTING  FEDERATE  /  CLUSTER: 

TYPE  OF  REQUEST:  RECONSTITUTION  /  MAGIC  KILL/  RESUPPLY  /  MAGIC 
MOVE 

BUMPER  NUMBER(S)  FROM  REQUESTING  CLUSTER: 

BUMPER  NUMBER(S)  OF  "OTHER"  UNIT: 

DTG  SENT  TO  WHITE  CELL: 

DECISION: 

DTG  EXECUTED: 


Note  that  the  Buck  Slip  includes  the  date-time  group 
(DTG)  at  when  the  event  occurred  in  simulation  time 
which  may  be  different  from  clock  time.  This  is  to 
make  the  audit  process  easier  for  analysts  whose  tools 
generally  record  in  simulation  time.  Once  completed, 
the  buck  slip  is  given  to  the  Node  Boss  of  the 
simulation  cluster  which  represents  the  unit  of  the 
simulation  interactor  or  military  role  player.  The  Node 
Boss  reviews  it  for  completeness  and  gives  it  to  the 
Federate  Battlemaster,  the  person  responsible  for  the 
proper  operation  of  the  simulation  at  each  Federate. 
The  battlemaster  will  also  review  it  for  logic  and 
completeness  and  will  submit  it  through  his  Federate 
Experiment  Control  to  the  Federation  White  Cell  where 
the  adjudication  team  works.  One  designated  person 
will  serve  as  the  full  time  recorder,  and  will  record  all 
adjudication  requests  and  actions  taken.  As  the  requests 
are  received  the  recorder  will  log  the  request  and  enter 
the  information  into  a  spreadsheet.  The  following  table 
(Table  1)  was  used  during  JFEWE. 


Figure  4.  Adjudication  Request  “Buck  Slip” 


Table  1.  JFEWE  Adjudication  Log  with  Sample  Entries  (All  Times  MST) 


Serial 

DTG  of  SIM 

Event 

Requesting 
Federate  / 

Cluster 

Bumper  Number 
from 

Requesting  Cluster 

Bumper 
Number(s)  of 
'other’  Unit 

DTG  sent  to 
White  Cell 

Decision 

DTG  Executed 

090230  Mar  1 1 

2  MEB 

TF  HAMMER 

12SWG  6418  72327 

090230  MAR  1 1 

Ash  and  trash  only  (incl  2  x 
T72s).  Situated  there  at 
STARTEX  NFA  at  this 
point 

090230  Mar  1 1 

10  2316  MAR  11 

GORDON 

1 JA4FATCN01 B 

1 JA4FATCN01 B 

1 J7DIVTRT01 B 

1 JACAVTCN01 B 

102316  MAR  11 

SIGNAL  TAC  COMMS 
NODE  X  2.  Links  with 

Serial  15.  Federate 

B'Master  Ft  Benning  liasing 
with  DES  at  Monroe  to 
regenerate 

102316  MAR  11 

100014  MAR  11 

RUCKER 

AH1Z/9  4828972672 

N/A 

100014  MAR  11 

COBRA  crashed  due  to 
operator  being  away  from 
stn  (due  to  TORNADO 
warning  yesterday). 

Request  to  regenerate 
made  via  Fed  BM. 

1 00036  Mar  1 1 

5 

100005  MAR  11 

IBCT  A-CO  2-44 

UNIT  UNKNOWN 

1BRZACWM66XR 

10020  MAR  11 

Entity  been  engaged  with 
5.56  and  7.62  Magic  Kill 
requested  via  Federation 
battlemaster 

100030  MAR  11 

This  spreadsheet  will  become  critical  during  post 
mission  analysis  as  it  will  help  reconcile  numerical 
analysis  questions  such  as  lethality  and  survivability. 
For  example,  if  during  a  battle,  10  tanks  and  their 
crews  were  destroyed,  but  through  adjudication  4  tanks 


and  crews  were  reconstituted,  it  is  possible  that  those  4 
tanks  and  crew  might  be  destroyed  in  later  fighting. 
Depending  on  the  simulation  and  how  data  is  logged,  it 
might  record  a  total  of  14  kills,  double-counting  the 
four  tanks  that  were  regenerated  as  a  result  of  the 
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adjudication.  Through  detailed  record  including  the  use 
of  timestamps  and  locations  for  the  requests,  analysts 
can  conduct  their  data  analysis  more  accurately.  It  is 
important  to  rehearse  the  flow  of  information,  and  it  is 
recommended  that  it  be  practiced  during  the  Pilot  or 
Trial  Runs  of  the  experiment. 

During  JFEWE,  both  a  Federation  Damage  Effects 
Server  (DES)  and  High  Level  Architecture  (HLA) 
Logger  were  used.  The  DES  provided  a  single  point  of 
adjudication  for  all  engagements  within  the  federation. 
The  HLA  Logger  recorded  all  HLA  traffic  across  the 
network.  Both  were  important  tools  to  investigate 
validity  of  adjudication  requests.  Through  examining 
their  logs,  the  White  Cell  and  Simulation  Control 
validated  interactions  or  engagements  and  used  them  to 
piece  together  information  from  what  was  reported  by 
Blue  and  Red  role  players  and  simulation  interactors. 
Although  DES  could  potentially  serve  as  a  standalone 
system  of  “instant  replay”,  it  ultimately  was  seen  as  a 
single  point  of  failure  and  the  HLA  was  used  as  a 
redundant  system.  Even  with  these  tools  it  is  critical  to 
keep  the  log  shown  above  to  have  a  single  point  of 
reference  of  adjudications  and  to  ensure  adjudications 
are  completed  correctly.  For  example,  if  the 
regeneration  of  an  entire  infantry  squad  was  approved, 
but  when  humans  completed  the  regeneration  they 
brought  an  entire  platoon  back  to  life,  there  would  be 
written  confirmation  that  the  regeneration  should  have 
been  a  squad,  not  a  platoon.  During  JFEWE, 
adjudications  were  completed  remotely  by  each 
experiment  federate  (in  some  simulations,  even  by 
cluster  coordinators).  The  log  accounted  for  authorized 
versus  unauthorized  adjudications.  Examples  of 
unauthorized  adjudications  could  be  a  participant  who 
resupplies  his  units  with  ammunition  or  fuel  to  speed 
up  the  process  quicker  than  it  would  happen  in  real  life 
or  resurrection  of  key  personnel  or  equipment  that  had 
been  legitimately  killed  by  the  enemy.  In  events  which 
a  DES  or  HLA  logger  is  not  available  or  practical,  the 
adjudication  log  is  essential. 

Step  5:  Implement  the  Adjudication  Procedures 

Although  it  may  appear  that  proper  execution  of  the 
previous  steps  would  make  implementation  an  easy 
process,  it  is  always  difficult.  An  umpire  or  referee  has 
a  rule  book  to  follow,  but  must  ultimately  make  the 
right  call.  Experience  shows  that  some  adjudications 
may  be  easy  decisions.  Examples  include  known 
munitions  errors,  terrain  problems  that  have  been 
identified  prior  to  execution,  and  outright  cheating  on 
the  part  of  participants.  On  the  other  hand,  most  require 
research  and  analysis  on  the  part  of  the  Adjudication 
Team.  Additionally,  the  decisions  must  be  expedient 
to  avoid  a  compounding  of  problems  or  errors.  This 


requirement  for  expediency  many  times  demands  fairly 
fast  judgment  calls  to  avoid  falling  behind  in  the 
process,  while  at  the  same  time  maintaining 
consistency.  Redundancy  must  exist  among  the  team 
members  and  back  up  personnel  must  be  identified  to 
fill  in  as  team  members  must  perform  other 
experimental  or  administrative  functions.  Also, 
contingencies  such  as  illness  or  other  absences  must  be 
taken  into  account  and  appropriate  personnel  identified 
to  fill  the  void.  A  system  should  also  exist  to  document 
new  adjudications  for  which  planning  was  not  done; 
these  will  be  added  to  the  procedures  for  the  next 
experiment. 

Step  6:  Conduct  After  Action  Review  and  Update 
Procedures 

During  the  experiment,  it  is  critical  to  conduct  daily  hot 
washes,  known  as  after  action  reviews,  for  several 
purposes.  First,  it  is  critical  to  ensure  all  team  members 
review  the  day’s  adjudications  and  ensure  that  all 
documentation  is  complete  before  ending  that  day's 
activities.  Generally  speaking,  most  days  start  off  with 
addressing  new  administrative  issues  and  consequently 
there  is  no  time  at  the  beginning  of  each  day  for  a 
review  of  the  previous  day’s  adjudications.  Also,  the 
information  is  still  fresh  in  all  participants’  minds  if 
follow  up  is  required  to  capture  all  of  the  data.  Second, 
it  is  important  to  review  what  went  well  and  what  went 
poorly  as  the  team  conducted  the  procedures.  Any 
changes  to  the  procedures  should  be  documented,  but 
no  changes  should  be  implemented  in  the  middle  of  the 
experiment  without  consultation  with  all  interested 
parties.  This  may  include,  but  is  not  limited  to:  the 
Experiment  Director,  the  customer  or  experiment 
sponsor,  the  Lead  Analyst,  and  all  affected  federates.  If 
approval  is  granted  to  change  the  procedures,  then  the 
process  should  be  formalized  and  all  simulation 
interactors,  military  role  players  (friendly  and  enemy), 
and  technical  support  personnel  such  as  the 
Battlemasters  and  Node  Bosses  must  be  informed  of 
the  change.  If  the  change  must  wait  until  the  next 
experiment,  the  change  should  be  made  and  briefed 
during  the  experiment  planning  conferences  during  the 
experiment  planning  and  preparation  phases. 

Use  of  the  Adjudication  Procedures  During  JFEWE 

The  Army  Experimentation  Community  of  Practice 
(CoP)  [depicted  on  the  next  page  in  Figure  6] 
successfully  planned,  coordinated,  and  executed  the 
JFEWE  experiment  from  June  2010  to  March  2011. 
During  that  time,  the  CoP  conducted  four  seminars  and 
three  planning  conferences  to  develop  the  Data 
Collection  Master  Plan  (DCMP),  analysis,  tactical 
scenario,  training,  simulation,  integration,  resource, 
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and  manning  plans.  MBL  conducted  the  FE  Seminar 
(2-3  June  2010)  to  inform  the  CoP  on  current  and 
future  capabilities  and  requirements  to  conduct  FE 
operations  against  a  “near  peer”  threat  with  anti-access 
and  area  denial  capabilities  to  begin  developing  the 
experiment  construct,  to  achieve  experiment  objectives 
identified,  and  to  codify  organization  support  and 
requirements. 

As  summarized  in  the  JFEWE  Final  Report,  the  CoP 
successfully  conducted  the  JFEWE  simulation  exercise 
(SIMEX)  from  28  February  to  17  March  2011  and 
concluded  with  a  Senior  Leader  Seminar  (SLS)  to 
ARCIC  Director  on  18  March  2011.  Participants  used 
an  approved  Multi  Level  Scenario  (MLS).  The 


operational  environment  consisted  of  the  full  range  of 
combat  operations  consistent  with  tactical  operations 
during  2018  time  frame  against  a  “near  peer”  enemy 
with  anti-access  and  aerial  denial  capabilities.  During 
the  execution  of  the  SIMEX,  nine  organizations 
provided  the  distributed  environment  via  the  Battle  Lab 
Collaborative  Simulation  Environment  (BLCSE) 
network,  which  integrated  five  simulation  models  and 
various  Command  and  Control  Systems. 

The  following  figure  lists  specific  models,  simulations 
and  supporting  tools  used  at  the  various  federates. 
Preparation  of  a  similar  reference  is  critical  for 
completion  of  Steps  1  and  2  of  the  Adjudication 
Process  and  helps  to  identify  participants  in  Steps  3-6. 


Models  &  Simulations  Federation 


MBL-  Fort  Benning 

Models,  Tools  and  Systems 

OneSAF 

ACRT-GV 

CPOF 

TACWEB 

HLA  Logger 

JVTR 

AsTI 

Representation: 

Blue  Maneuver  Units  (1-44,  2-44, 
3-44  IN  BNs,  A  Co,  1-5  SBCT,  3-4 
CAV,  R&S  Bde,  BSB  &  MEB) 

USN  Carrier  Strike  Group 
USMC  MEB 
Blue  UAS  (CLI) 

Blue  Mines 
Sensor  Control 
C2  Snapshots 
lxSOCCE  (ARSOCIC) 

EXCON  &SIMCON 

Red  Maneuver 

Red  Fires 

Red  UAS 

lEDs 

COBs 

MCBL  -  Leavenworth 

Models,  Tools  and  Systems 

CPOF 

CPOF  Server 

JVTR 

AsTI 

Representation: 

JTF,  JFLCC,  JFACC,  &  JFMCC 
7th  Div  with  JAGIC  cell 
Sustainment  and  Medical  Bde 


USAF  -  Hurlburt 

Models,  Tools  and  Systems 

Air  Warfare  Simulation  (AWSIM) 

Representation: 

USAF  Fixed  Wing 


Fires  BL-  Fort  Sill 

Models,  Tools  and  Systems 

FireSIM 
FireSIM  GUI 
AFATDS 
EADSIM 
RTOS 

Representation: 

Blue  Artillery  (Fires  Bde  &  Bn) 
Naval  Gun  Fire 
ADA  &  Counter  Radar 
ADA 


ARSOCIC  -  Fort  Bragg 

Models,  Tools  and  Systems 

CPOF 

JVTR 

AsTI 

Representation: 

JSOTF/1  x  SOCCE 
4IBCT Response  Cell 


SMDC  -  Redstone 

Models,  Tools  and  Systems 

EADSIM 

JEMS 

GCCS-A 

Representation: 

Space  &  High  Altitude  systems 


Aviation  EAE  -  Fort  Rucker 

Models,  Tools  and  Systems 

ATCOM 

ACRT-AV 

CPOF 

Representation: 

Blue  Rotary  Wing  Air  (AVN  Bde) 

Blue  ERMP  UAS 


NOSC,  NEAE  -  Fort  Gordon 

Models,  Tools  and  Systems 

OneSAF 

CES 

CRA-P 

Network  Timeserver 
Representation: 

COMMS  Effects 

S-6s supporting  each  Maneuver  HQ 

MSBL-  Ft  Leonard  Wood 

Models,  Tools  and  Systems 

OneSAF 

CPOF 

JVTR 

AsTI 

Representation: 

7th  MEB 
BSTB  &  BEB 


SustBL-  Fort  Lee 

Models.  Tools  and  Systems 

OneSAF 

JDLM  (ran  on  stand  alone  RTI  at  Ft.  Lee) 
CPOF 

Representation: 

Sustainment  Support  elements 


TICC  South 

Models,  Tools  and  Systems 

OneSAF 
Back  up  DES 
Back  up  RTI 
Function: 

Software  Engineering  Support 

STTC  -  Orlando 

Models,  Tools  and  Systems 

OneSAF 

Function: 

Software  Engineering  Support 

ARCIC  BLCSE -Fort 
Monroe 

Models,  Tools  and  Systems 

Starship/Stargen 
RTI  Control 
Function: 

Technical  Operation  and 
Control 

NVESD 

Models,  Tools  and  Systems 

MTS 

Universal  Controller  (UC) 

CMS2 

CMS 

Snap  Server 

NVIG 

Function: 

Night  Vision  Toolset  Support 

Legend  i 

Friendly  Forces  Civilians  1 
_  _  _  _  Enemy  Forces  _  __  _  J 


Figure  6.  Models,  Simulations,  and  Support  Tools 


LESSONS  LEARNED-  The  following  are  areas  of 
the  Adjudication  Procedures  which  needed 
improvement  during  or  as  a  result  of  JFEWE 

Leaders  must  enforce  the  progression  of  requests. 

At  the  MBL,  as  simulation  interactors  and  role  players 


became  more  familiar  with  the  experiment  construct 
and  how  adjudications  decisions  were  made,  they  short 
circuited  the  process  and  went  straight  to  the 
simulation  control  or  white  cell  with  their  requests. 
This  left  the  node  boss  and  cluster  coordinator  out  of 
the  process  and  they  may  have  had  input  (possibly 
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beneficial)  from  a  technical  standpoint  that  could  not 
be  entered  into  the  decision  process.  Additionally,  this 
causes  practical  problems  when  individuals  are 
entering  the  work  area  of  the  White  Cell  to  argue  their 
case.  They  obviously  are  not  at  their  assigned  duty 
station  as  the  simulation  run  continues  and  there  may 
be  information  such  as  MESLs  or  enemy  disposition 
that  role  players  should  not  see.  It  is  important  in  all 
venues  to  review  the  established  adjudication 
procedures  to  ensure  they  are  responsive  and  provide 
feedback  to  the  experiment  participants.  It  is  equally 
important  to  enforce  those  procedures  during 
experiment  execution. 

All  personnel  must  be  consistent  in  the  completion 
of  documents  (logs  and  buck  slips).  Even  with  the 
best  tracking  tools  in  place,  discipline  is  required  in  the 
implementation  to  ensure  all  documents  are  completed 
in  their  entirety  and  with  as  much  detail  that  is  known 
at  the  time  of  the  report.  Many  of  the  buck  slips 
submitted  during  the  execution  of  JFEWE  were 
incomplete  or  did  not  make  sense  to  the  reader.  The 
completed  buck  slip  should  leave  no  doubt  as  to  who 
was  involved  and  what  the  requested  adjudication  is. 
The  buck  slips  feed  into  the  adjudication  log  and  the 
same  principles  hold  true  for  completing  it,  it  must  “tell 
the  story,”  account  for  what  actions  were  taken  (or 
not),  and  a  justification  for  the  actions  taken.  Analysts 
will  refer  to  the  log  to  ensure  proper  accounting  for 
entities  or  supplies  that  were  brought  back  or 
reconstituted  as  the  result  of  adjudication.  In  the  case 
of  JFEWE,  an  area  in  which  to  improve  in  the  future 
would  be  the  narrative  remarks  to  spell  out  exactly 
what  happened  and  a  justification. 

Adjudication  decisions  must  be  evident  to  all 
federates.  During  JFEWE,  there  was  no  mechanism  to 
inform  all  federates  (the  EXCONs)  of  adjudication 
decisions.  All  of  those  who  were  involved  in  a  specific 
adjudication  were  informed,  but  there  was  not  a 
SharePoint  or  distributed  tracker.  In  many  cases,  the 
method  used  was  sufficient  and  informed  all  necessary 
parties.  However,  there  are  many  instances  either  for 
necessity  or  for  informational  purposes  that  all 
federates  should  have  access  to  the  information.  One 
example  would  be  if  enemy  air  defense  systems  were 
destroyed  by  friendly  artillery  and  then  restored  as  the 
result  of  adjudication.  Not  only  would  the  enemy 
forces  and  the  friendly  USAF  role  players  need  this 
information  (otherwise  they  would  be  operating  under 
the  premise  that  the  enemy  air  defense  is  weaker  than  it 
really  is),  but  the  USMC  and  Army  role  players  will 
also  need  that  information.  The  information  would  also 
be  important  for  intelligence  analysts  who  may  be 
tracking  enemy  air  defense  strength  system-by-system. 


LESSONS  LEARNED-  The  following  are 
procedures  to  sustain  or  that  validated  the  use  of 
the  adjudication  procedures. 

The  adjudication  procedures  allowed  for 
experiment  execution,  even  though  several 
simulations  had  integration  problems.  Due  to 

various  reasons  beyond  the  scope  of  this  paper,  the 
effects  of  some  simulations  were  not  visible  or  had 
significant  delay  in  appearing  in  other  simulations.  For 
example,  friendly  Air  Force  aircraft  would  engage 
enemy  personnel  or  weapons  systems  but  would  not  be 
able  to  determine  if  their  attack  was  successful  or  not, 
such  as  when  the  simulation  portraying  the  friendly 
aircraft  did  not  model  individual  combatants  (Soldiers, 
insurgents,  civilians)  or  obscurants  (the  coverage  of 
buildings).  Through  the  use  of  the  adjudication 
procedure,  the  adjudication  process  was  followed  and 
feedback  was  provided  to  the  Air  Force  experiment 
control  members.  Additionally,  a  target  which  was 
attacked  and  would  have  suffered  damage  under  real 
conditions  was  appropriately  damaged  or  destroyed 
using  the  adjudication  procedures.  Although  the 
adjudication  procedures  were  originally  designed  to 
handle  issues  that  would  arise  on  a  case  by  case  basis, 
the  procedures  also  served  well  for  systemic 
shortcomings  in  models,  allowing  for  an  efficient  and 
predictable  workaround. 

Documentation  must  exist  for  all  human 
interventions  and  overrides  of  the  simulations.  The 

purpose  of  the  adjudication  log  is  to  provide  the  analyst 
team  a  record  of  human  intervention  into  the 
simulation  results.  Consequently,  it  serves  as  a 
practical  tool  to  document  any  intervention,  including 
those  from  the  Master  Scenario  Event  List  (MSEL)  (as 
defined  below)  or  if  changes  were  made  to  the 
simulation  conditions  to  achieve  experiment  objectives. 
For  example,  if  an  enemy  unit  is  rendered  combat 
ineffective  through  combat,  its  presence  may  be 
necessary  so  that  the  friendly  forces  can  continue  to 
operate  in  order  to  gain  insights  into  their  operations. 
In  this  case,  the  Experiment  Director  may  determine 
that  even  though  the  loss  was  through  logical  and 
realistic  engagements,  the  enemy  units  should  be 
regenerated.  In  this  example,  using  the  adjudication  log 
is  a  practical  method  to  ensure  that  the  human 
intervention  is  accounted  for  during  post-experiment 
analysis.  As  mentioned  earlier  in  the  development  of 
procedures  discussion,  it  also  ensures  that  the 
adjudications  were  completed  correctly,  so  that  a 
platoon  was  not  resurrected  when  only  a  squad  should 
have  been. 
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The  adjudication  procedures  were  nested  closely 
with  actions  injected  by  the  White  Cell  from  the 
Master  Scenario  Event  List,  providing  more  realism 
for  the  participants  and  better  tracking  by  the 
analysis  team.  The  purpose  of  the  MSEL  cell  is  to 
ensure  execution  of  events  that  may  not  occur  through 
the  natural  events  of  the  experiment  runs.  For  example, 
if  the  medical  community  requires  a  mass  casualty  to 
study  the  ability  of  friendly  forces  to  conduct  large 
scale  medical  evacuation,  it  is  very  possible  that  no 
single  battle  produces  a  mass  casualty  event.  In  that 
case,  the  MSEL  team  would  establish  a  time  and 
location  to  introduce  a  mass  casualty  event  among 
friendly  forces.  Although  closely  related  to  the 
adjudication  cell,  the  MSEL  cell  is  a  separate  group, 
but  must  closely  coordinate  their  actions  so  detailed 
records  are  maintained  and  the  experiment  does  not 
become  desynchronized.  During  JFEWE,  the  two  cells 
were  co -located  and  their  coordination  and  knowledge 
of  each  other’s  actions  facilitated  experiment 

execution. 

CONCLUSIONS  AND  RECOMMENDATIONS 

The  JFEWE  validated  the  use  of  adjudication 
procedures  as  a  means  to  account  for  model  and 
simulations  shortcomings  and  facilitate  experiment 

White  Cell  actions.  Of  the  eight  Experiment  Objectives 
identified  earlier,  the  adjudication  procedures 

facilitated  accomplishment  of  three:  Step  1)  Focus  on 
the  operational  problem  of  Anti-Access  with  a 
significant  tactical  force  (Multiple  BDEs/DIV  force). 
Step  2)  Critically  explore  and  assess  concept 
supporting  ideas  and  required  capabilities  in 

challenging  operational  environments  to  revise  Army 
concepts  to  drive  capability  development  in  the  2018- 
2026  timeframe,  and  3)  Conduct  the  experiment  within 
a  joint  context/problem  to  inform  and  exercise 
JFCOM’s  draft  JOA  Concept  as  they  allowed  for 
consistent  employment  of  friendly  fixed  wing  aircraft 
employment  and  effects,  as  well  as  effects  of  enemy 
anti-access  weapons  against  friendly  Joint  assets 
(Naval  and  Air).  As  demonstrated  in  this  paper,  it  is 
critical  to  clearly  establish  how  and  when  the 
procedures  are  implemented  for  each  experiment,  who 
will  be  involved  in  the  decision  concerning  the 
adjudication  and,  most  importantly,  how  the  approvals 
and  denials  of  the  adjudication  requests  will  be 
documented.  Once  an  initial  draft  of  procedures  is 
designed,  it  is  important  to  staff  them  with  all 
stakeholders  to  ensure  they  are  complete,  flow 
logically  and  are  easily  understood  by  expected 
participants. 

Several  topics  discussed  in  this  paper  lend  themselves 
to  future  research  and  documentation  by  modeling  and 


simulation  professionals.  Perhaps  the  most  obvious 
topic  for  further  investigation  comes  from  the  opposite 
mind-set;  that  during  an  experiment,  models  and 
simulations  should  run  in  whatever  state  they  exist  and 
humans  should  not  override  them.  Another  topic  for 
research  may  be  the  development  of  an  electronic 
system  that  spans  from  request  to  decision  and  post 
experiment  analysis,  which  would  benefit  both  the 
experimentation  and  training  community  of  practice. 
Research  into  the  analysis  of  these  or  similar 
adjudication  procedures  in  a  more  data-centric 
constructive  simulation  venue  such  as  a  material  or 
procurement-  oriented  investigation  would  provide 
useful  insights  or  guidelines  that  might  be  different 
from  a  venue  like  JFEWE  which  is  more  concept- 
focused.  In  closing,  it  is  recommended  that  the 
different  communities  of  practice  consider 
documentation  and  sharing  of  internal  adjudication 
procedures  and  leverage  existing  SharePoints  and 
knowledge  management  systems  to  facilitate 
information  exchange.  Open  collaboration  and  sharing 
of  lessons  learned  through  experience  and  refinement 
will  benefit  all  agencies  involved  in  constructive  and 
virtual  simulation  events. 
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